Parallel entropy coding

ABSTRACT

Parallel coding of digital pictures is described. A digital picture is divided into two or more vertical sections. Two or more corresponding Stage  1  encoder units can perform a first stage of entropy coding on the two or more vertical sections on a row-by-row basis. The entropy coding of the vertical sections can be performed in parallel such that each Stage  1  encoder unit performs entropy coding on its respective vertical section and returns a partially coded Stage  1  output to a Stage  2  encoder unit. Each partially coded Stage  1  output includes a representation of data for a corresponding vertical section that has been compressed by a compression factor greater than 1. The Stage  2  encoder unit can generate a final coded bitstream from the partially encoded Stage  1  output as a Stage  2  output.

FIELD OF THE INVENTION

Embodiments of the present invention are related to video encoding andmore particularly to parallel encoding of digital pictures.

BACKGROUND OF THE INVENTION

Digital signal compression is widely used in many multimediaapplications and devices. Digital signal compression using acoder/decoder (codec) allows streaming media, such as audio or videosignals to be transmitted over the Internet or stored on compact discs.A number of different standards of digital video compression haveemerged, including H.261, H.263; DV; MPEG-1, MPEG-2, MPEG-4, VC1; andAVC (H.264). These standards, as well as other video compressiontechnologies, seek to efficiently represent a video frame picture byeliminating or reducing spatial and temporal redundancies within a givenpicture and/or among successive pictures. Through the use of suchcompression standards, video contents can be carried in highlycompressed video bit streams, and thus efficiently stored in disks ortransmitted over networks.

MPEG-4 AVC (Advanced Video Coding), also known as H.264, is a videocompression standard that offers significantly greater compression thanits predecessors. The H.264 standard is expected to offer up to twicethe compression of the earlier MPEG-2 standard. The H.264 standard isalso expected to offer improvements in perceptual quality. As a result,more and more video content is being delivered in the form of AVC(H.264)-coded streams. Two rival DVD formats, the HD-DVD format and theBlu-Ray Disc format support H.264/AVC High

Profile decoding as a mandatory player feature. AVC (H.264) coding isdescribed in detail in ISO/IEC 14496-10:2009, “Informationtechnology—Coding of audio-visual objects—Part 10: Advanced VideoCoding, Edition 5” May 13, 2009, which is incorporated herein byreference. A copy may be downloaded at the following URL:http://www.iso.org/iso/iso_catalogue/catalogue_tc/catalogue_detail.htm?csnumber=52974.

Video signal coding tends to be a computationally intensive applicationrequiring a high memory bandwidth. Multi-processor systems have beendeveloped with high computing performance and relatively lower powerconsumption. Some multi-processor systems have dedicated local memoryassociated with each processor core. It is appealing to implement videoencoding on a multi-processor platform since the memory bandwidth insuch systems may scale with the number of processors. However, due tothe complexity of the encoding process it is difficult to optimizeparallel video encoding for multi-processor platforms having more thantwo processor cores.

The video encoding process removes spatial and temporal redundanciesbetween and within video pictures. However, this process can create datadependencies among video pixels. When encoding a video stream inparallel, these data dependencies tend to slow down the encoding processor lower the quality of the video stream being encoded. For example,encoding multiple sections of a given video picture in parallel mayrequire each section to use data from another section. The required datamay be momentarily unavailable if the section is currently beingprocessed. The data dependency issue may be resolved by slowing down theencoding process, such that sections are sequentially encoded to avoidmissing data dependencies. However, this may cause very large encodingdelays. The data dependency issue may also be resolved by creatingartificial data isolations to fill in for the currently inaccessibledata dependency. This, however, may reduce the encoding quality of thevideo stream.

It is within this context that embodiments of the present inventionarise.

BRIEF DESCRIPTION OF THE DRAWINGS

The teachings of the present invention can be readily understood byconsidering the following detailed description in conjunction with theaccompanying drawings, in which:

FIG. 1 is a flow diagram illustrating an example of parallel videoencoding in accordance with an embodiment of the present invention.

FIG. 2 is a block diagram illustrating an example of a parallel entropycoding system and method in accordance with an embodiment of the presentinvention.

FIGS. 3A-3C are flow diagrams illustrating different possibilities forimplementing a second stage of entropy coding in a CABAC implementationaccording to alternative implementations of certain embodiments of thepresent invention.

FIG. 4 is a flow diagram illustrating a possible implementation ofhandling subsection skip run computation for entropy coding in a CAVLCimplementation according to certain embodiments of the presentinvention.

FIG. 5A is a flow diagram illustrating a first possible implementationof handling subsection QP difference encoding in entropy coding in aCAVLC implementation according to certain embodiments of the presentinvention.

FIG. 5B is a flow diagram illustrating a first possible implementationof handling subsection QP difference encoding in entropy coding in aCABAC implementation according to certain embodiments of the presentinvention.

FIG. 6A is a flow diagram illustrating a second possible implementationof handling subsection QP difference encoding in entropy coding in aCAVLC implementation according to certain embodiments of the presentinvention.

FIG. 6B is a flow diagram illustrating a second possible implementationof handling subsection QP difference encoding in entropy coding in aCABAC implementation according to certain embodiments of the presentinvention.

FIG. 7 illustrates a block diagram of an example of an encoder unit thatmay be used to implement one or more stages of parallel entropy codingin accordance with an embodiment of the present invention.

FIG. 8 illustrates an example of a CELL processor that may be apparatusthat may be used to implement one or more stages of parallel encoding inaccordance with an embodiment of the present invention.

FIG. 9 illustrates an example of a computer-readable storage mediumencoded with computer readable instructions for implementing parallelencoding in accordance with an embodiment of the present invention.

DESCRIPTION OF THE SPECIFIC EMBODIMENTS

Although the following detailed description contains many specificdetails for the purposes of illustration, anyone of ordinary skill inthe art will appreciate that many variations and alterations to thefollowing details are within the scope of the invention. Accordingly,the exemplary embodiments of the invention described below are set forthwithout any loss of generality to, and without imposing limitationsupon, the claimed invention.

Introduction

The video encoding process generally involves breaking down a pictureinto multiple sub-sections, performing a mode search to determinewhether to use inter-prediction or intra prediction, followed by aprocess known as entropy coding. Entropy coding is a highly sequentialprocess.

Entropy coding is zero/one based and highly sequential. As used herein,“zero/one based” has two meanings. First, Entropy coding tries to encodeeach symbol with a minimum number of 0s or 1s. In one entropy codingimplementation known as context adaptive variable length coding (CAVLC),the 0s or 1s are called bits. In another entropy coding implantationknown as context adaptive binary arithmetic coding (CABAC), the 0s or 1sare called bins. The CABAC coding compresses these bins into bits withcontent adaptive arithmetic coding. Both implementations of the Entropyencoding process involve data dependencies down to bit level, whichmakes the process highly sequential.

At a lowest level, entropy coding is “bit-based” meaning that a codingdecision for a given bit depends on the result for a previous bit. Forsome codecs, the coding mechanism for each symbol is adaptive, meaningthat the entropy coding tries to encode each symbol with a minimumnumber of bits. Such coding schemes attempt to code more probablesymbols with a lesser number of bits. In some codecs, such as AVC,binary decisions are separate from final stream generation. In suchcodecs, binary decisions as to whether a bit will be coded as a zero ora 1 may be made based on video content. In addition, certain codingtechniques, such as context adaptive binary arithmetic coding (CABAC),implement a further level of coding of bits. These various codingschemes place a large bandwidth requirement for implementing entropycoding on a single CPU core processor. Furthermore, the entropy codingprocess is not scalable.

Although AVC entropy coding is a sequential coding process, theinventors of the current invention have recognized that there arenon-sequential parts of the process that can be localized andimplemented in parallel.

A generalized parallel encoding process is described incommonly-assigned U.S. patent application Ser. No. 12/553,073, filedSep. 2, 2009, the entire contents of which are incorporated herein byreference.

FIG. 1 is a flow diagram illustrating an example of parallel videoencoding 100 that deals with the issue of data dependency according toan embodiment of the present invention. As used herein, parallel videoencoding generally refers to encoding different sections of a videopicture or different video pictures in such a way that process ofencoding the respective sections or pictures overlap in time. In theprophetic example illustrated in FIG. 1, a controller 101 (e.g., adigital computer processor) initially chooses an encoder unit (labeledE0) from among a plurality of such units available in a processingsystem to act as the master/server encoder for the task of encoding oneor more video pictures. The controller 101 also determines how manyadditional encoder units are needed as clients/slaves of themaster/server encoder E0 to perform parallel encoding as indicated at103. The client/slave encoder units are labeled as E1 . . . EN, where Nrepresents the total number of client/slave encoders that performencoding in parallel with the master/server unit E0. The master serverencoder unit E0, and client/slave encoder units E1 . . . EN may beimplemented in the form of digital co-processors that may be lesscomputationally intensive devices than the controller 101. Each suchco-processor may optionally include an associated local storage that theco-process may quickly access. The local storage may store codedinstructions that the co-processor can use to implement a part of thetask of encoding a picture. The local storage may also store data to beprocessed by the co-processor by executing the stored instructionsand/or data generated as a result of executing the stored instructions.In an alternative embodiment, each encoder unit may be an individualdigital computer processor, e.g., a parallel processing system such as aCELL processor.

By way of example, the controller 101 may select encoder unit E0 as themaster/server. The controller 101 may send encoder unit E0 a taskpackage which may include encoded information that the master/serverencoder E0 may use to determine where to find the video stream, how toencode the video stream, and how to categorize the content of the videostream. In determining how to encode the video stream, the followinginformation may be included in the task package: 1) informationregarding whether the bit rate should be high or low; 2) standardconstraints including: the decoder level to be associated with the videostream, e.g., whether a simple or complex decoder will be used at thereceiving end, the memory usage associated with the video stream, and aprediction mode associated with the video stream; and 3) featureconstraints including: whether the video stream is associated with afast playback mode or not, how fast the playback associated with thevideo stream is, and the device on which the video stream is to beplayed back. The categorization of content may include the color codingused (e.g., RGB or YUV) or a color matrix used to convert the videostream from one color code to another.

After the master/server encoder E0 receives the task package, it maybegin the encoding process for a given video stream. The encodingprocess will typically begin with the first video picture of the videostream, but for the sake of example, the video encoding process isdescribed below for an arbitrary video picture 105. The master/serverencoder E0 divides the video picture 105 into N+1 vertical sections thatcorrespond to the N+1 encoders available for parallel processing, asindicated at 107. Thus, in the present example, E0 is assigned sectionS0, E1 is assigned section S1, and so on and so forth up until theN^(th) encoder unit EN, which is assigned section SN.

Each vertical section S0, S1, . . . SN includes data representing aportion of the complete video picture 105. Each vertical sectionincludes at least one complete column of subsections of the picture.Examples of subsections include, but are not limited to individualpixels, blocks (4×4 groups of pixels), sub-macroblocks (8×8 groups ofpixels), and macroblocks (MB) (16×16 groups of pixels). As used herein,a complete column refers to a column that extends from a top of thepicture to a bottom of the picture. By way of example, and not by way oflimitation, one complete column of macroblocks would be portion of apicture 1 macroblock wide by M macroblocks tall, where M is the numberof rows of macroblocks in the picture.

In certain embodiments, the mode search may be performed in parallel bythe encoder units E0, E1, . . . , EN. Each encoder unit E0, E1, . . . ,EN can perform a mode search on its respective section S0, S1, . . . SNto determine whether each subsection (e.g., each MB) of the respectivesection should be inter-coded or intra-coded. This can include a motionsearch for a best inter-prediction match, an intra search for a bestintra-prediction match, and an inter/intra comparison to decide how theMB should be coded. The master/server unit E0 begins by doing a modesearch for the first row of macroblocks in order to determine whethereach macroblock should be inter-coded or intra-coded as described at109.

The master/server encoder E0 sends the search result of the right mostMB and a proposed prediction mode of the left most MB in the rightneighbor's section of each row to unit E1. Consequently, unit E1 has towait for unit E0's right most MB mode search result and proposedprediction mode for E1's left most MB before unit E1 can start its modesearch of this MB row. For the same reason, unit E2 waits for unit E1'sright most MB result and proposed prediction mode for E2's left most MB,and so on. As the result, each encoder unit starts one MB row later thanthe unit handling its left neighbor section.

Because MB mode search and entropy encoding of a MB depend on its upperright neighbor MB, to finish the right most MB prediction search anencoder unit for a section that has a right neighbor needs theprediction mode of the left most MB in the right neighbor section. But,in this encoder implementation, the data dependencies from a rightsection to its left neighbor are not allowed. To solve this problem, theencoder unit for the left section has to search for the prediction modeof the first MB in its right neighbor section. Because the predictionmay be done without knowing the correct upper right MB mode, the resultmay not be optimal. However, experiments have shown that this predictionmode is very close to the best. Then, the encoder unit for the rightneighbor section only can accept this prediction mode and use thisprediction mode to encode the left most MB in the right neighbor sectioninto the output bit stream. In this way it is possible to guarantee thatthe entropy encoding result will be correct with very small loss inquality.

The mode searches carried out by the different units may overlap intime. For example, to some extent, unit E1 may be involved in a modesearch for its (N−1)^(th) row of subsections as described at 111 whileunit EN is involved in a mode search for its first row of subsections asdescribed at 113. This mode search is then repeated for the next row ofsubsections in each vertical section until mode searching for the entirevideo picture has been completed.

The mode search may include a task known as motion compensation todetermine a mode result, e.g., whether intra-coding or inter-codingshould be used. Specifically, a mode search performed on a section mayproduce a motion vector (MV) and a transform coefficient that aresubsequently used along with one or more reference sections duringmotion compensation. The motion compensation may use these motionvectors and transform coefficients to describe the current section interms of the transformation of a reference section to a currentlyprocessing section.

As used herein, the term motion compensation generally refers to aprocess to build pixel predictors from a reference picture. By way ofexample, and not by way of limitation, in certain encoderimplementations motion compensation may be integrated with inter searchprocess. After inter search, the encoder may use the resulting motionvector to build pixel predictors. Then, the encoder may use the originalinput pixel and pixel predictor to calculate a prediction error referredto as residual pixels. A discrete cosine transform (DCT) may be used totranslate residual pixels into residual DCT coefficients. A processknown as quantization reduces the number of symbols used to representthese DCT coefficients. The resulting quantized DCT coefficients may beused by an entropy coding task.

Once it is determined whether the subsections in each row should beinter-coded or intra-coded, entropy coding may be performed. As usedherein, the term entropy coding generally refers to a task by whichprediction modes, motion vectors and quantized residual DCT coefficientsare translated into a compressed digital representation. The input ofentropy coding task is the mode search result. The output is acompressed digital representation of the mode search result.

To further optimize the encoding of a video stream, a method ofcompressing variable length symbols may be implemented within theentropy encoding task. The output of the entropy encoding task for agiven section of a picture may include a variable length coderepresenting DCT coefficients for the residual pixels in the givensection. The variable length code may be composed of multiple tokenseach of which represents a subsection of the variable length code. Thebit representation of these tokens are often compressed. Existingentropy coding techniques often use lookup tables to determine a bitrepresentation for each token. This can lead to computationalinefficiencies. In certain embodiments of the present invention entropycoding may be made more efficient by combining two or more tokenstogether. The resulting combination may be compared against a speciallygenerated lookup table to determine a bit representation for the tokencombination. Such embodiments are discussed in detail below with respectto FIGS. 6A-6B of U.S. patent application Ser. No. 12/553,073, filedSep. 2, 2009, which was incorporated by reference above.

The encoding process may include additional tasks. For example, withinthe mode search task, after mode search process, the encoder runs adecoder emulation process which is called encoder local decoding. Inencoder local decoding process, the encoder uses inverse quantization torecover residual DCT coefficients from quantized DCT coefficients. Then,it uses inverse DCT to get residual pixels (prediction error) fromresidual coefficients. Combining the prediction error with pixelpredictors, the encoder can get an uncompressed picture which should beexactly the same as the uncompressed picture generated by a standalonedecoder. The uncompressed picture may be sent to a de-blocking task forde-blocking. The de-blocking process may be used to average out pixelsat the block or MB boundaries. This is done to ensure that the encodedvideo picture fairly represents the original video picture. Afterde-blocking, the uncompressed picture may be saved and used as areference picture to do inter prediction for future pictures.

Both the de-blocking task and entropy coding task may start when themode search task is finished for one MB row. There is no timingdependency between de-blocking and entropy coding. However, as an extratiming constraint, the de-blocking task for one section may have to waitfor completion of de-blocking of each MB row on a neighboring sectiondue to data dependencies within the de-blocking task itself. Forexample, the de-blocking task on a given section may need to wait forthe de-blocking of its left neighbor to send the de-blocking result ofthe right most MB of the left neighbor to the encoder unit handling thegiven section.

Furthermore, for de-blocking, there may be a one MB column overlapbetween the encoder units for neighboring vertical sections. For a MBshared by encoder units for two adjacent sections, the encoder unit thatde-blocks the left section may perform vertical edge de-blocking on theshared MB and the encoder unit that de-blocks the right section mayperform horizontal edge de-blocking on the shared MB. By way of example,the de-blocking task for the macroblocks in the first column of sectionS1 may be shared between section S0's encoder unit E0 and section S1'sencoder unit E1. Encoder unit E0 may de-block the vertical edges of themacroblocks in the left most column of section S1 and encoder unit E1may de-block the horizontal edges of these macroblocks.

Referring again to the example illustrated in FIG. 1, the master/slaveencoder unit E0 begins by entropy encoding for each subsection in thecurrent row of section S0 as described at 115. Unit E1 maysimultaneously entropy encode each subsection in the current row ofsection S1 as described at 117. Unit EN may also simultaneously entropyencode each subsection in its current row of section SN as described at119.

At a certain point, enough of a given section is encoded such that thede-blocking process for that section may begin. For example, de-blockingof section S0 by the master/server encoder E0 is indicated at 121.De-blocking of section S1 by the encoder E1 is indicated at 123.De-blocking of section SN by the encoder EN is indicated at 125.De-blocking may be implemented in parallel on the encoder units E0 . . .EN or off-loaded to a different processor entirely. An example ofparallel de-blocking is described in commonly-assigned U.S. patentapplication Ser. No. 12/553,073, filed Sep. 2, 2009, the entire contentsof which have been incorporated herein by reference above.

Embodiments of the present invention solve the problem of how todistribute the task of entropy encoding a picture to multiple encoderunits and execute the task in parallel. In addition, embodiments of thepresent invention also address the problem of how to parallelize theencoding task within a slice.

One slice could cross multiple rows, or could even be smaller than onerow. For example, as specified in the AVC standard, one slice couldinclude any number of macroblocks from as few as 1 macroblock up to thenumber of macroblocks in a picture. Typically, the slice size can bedetermined by the user. The encoder can take the user assigned slicesize to do encoding. By way of example, and not by way of limitation, aslice may be a row of subsections, e.g., a row of macroblocks.

It is noted that although each slice of the picture can be encodedindependently, parallelizing entropy coding by assigning each slice to adifferent encoder unit is often impractical. This is because the numberof slices in a picture may vary and cannot be controlled by the systemdoing the encoding. Furthermore, embodiments of the present inventionaddress the problem of how data can be compressed before beingtransferred among encoder units, so that bandwidth and network delayscan be reduced for relatively low cost.

1) Parallel Entropy Encoding Scope:

Parallel entropy encoding generally applies to slice data encoding. Themajority of the entropy encoding task lies in slice data encoding, whichincludes encoding of all the subsection (e.g., macroblock) syntaxelements for that portion of a slice that lies within a given verticalsection. This task involves encoding the symbols that represent therelevant subsections into compressed symbols in the form of a bitrepresentation. This can compress the symbol by a factor of about 50-100depending on the quantization parameter used. Other syntax elements thatmay need to be encoded in certain coding standards, such as the AVCstandard, include sequence parameter set, picture parameter set, andslice header.

Additional compression may be done on the compressed symbols dependingon the coding standard used and the implementation details of thestandard that is used. For example, the AVC standard can implement twotypes of entropy encoding for slice data referred to as context adaptivevariable length coding (CAVLC) and context adaptive binary arithmeticcoding (CABAC). Both CAVLC and CABAC implementations include compressionof the symbols by a factor of about 50-100. For CABAC the 0-1probability (Binary Arithmetic Coding) for each symbol can be applied,which further compresses the compressed symbols by an additional factorof about 1.3:1.

2) Picture Partitioning for Parallel Entropy Encoding:

According to certain embodiments of the present invention, parallelentropy coding may be implemented as illustrated in FIG. 2. For parallelentropy encoding to work, a picture can be partitioned into multiplevertical sections, e.g., as described above with respect to FIG. 1. Theencoding task can be divided into two stages denoted herein as Stage 1and Stage 2. Stage 1 can be implemented in parallel by two or more Stage1 encoder units 201. By way of example, and not by way of limitation,the entropy encoder unit E0 . . . EN may be configured, e.g., bysuitable programming or equivalent hardware configuration, to performStage 1 of the entropy coding task on a corresponding one of thevertical sections S0 . . . SN. In Stage 1, each encoder unit E0 . . . ENcan encode and generate the bit representations of the symbols thatrepresent the relevant subsections of the corresponding sections S0 . .. SN. As noted above, this compresses the symbol data by a factor ofabout 50-100. The output of the Stage 1 entropy coding outputs 202 _(0,)202 ₁ . . . 202 _(N) for each of the vertical sections S0, S1, . . . SN,including the compressed symbol data for each of the sections can betransmitted from the corresponding encoder units E0, E1, . . . EN inpartially encoded bitstreams.

In Stage 2, a Stage 2 encoder unit 203 can form a final bitstream 204using the Stage 1 outputs 202 ₀ . . . 202 _(N) from each of the Stage 1encoder units 201. By way of example, and not by way of limitation, theStage 2 encoder unit 203 may be a dedicated one of the encoder units E0. . . EN, such as the master encoder unit E0. Alternatively, the Stage 2encoder unit 203 may be an entirely separate processor module other thanone of the encoder units E0 . . . EN. By way of example, and not by wayof limitation, for a CABAC implementation of AVC, the 0-1 probability(Binary Arithmetic Coding) for each symbol can be done during Stage 2.

Depending on the codec used, there can be data dependencies among thedifferent vertical sections S0 . . . SN for entropy encoding duringStage 1. For example, encoding a given MB may require some syntaxinformation from a previous MB in the same row as the given MB and/orsyntax information for a MB immediately above the given MB. Here theprevious MB can be the MB to the immediate left of the given MB or, ifthe given MB is the first MB in a MB row, the previous MB can be thelast MB in the MB row above the given MB. This means that encoding thefirst MB column of a vertical section can be dependent on the MB syntaxinfo from the last MB column in its left neighbor vertical section.

Based on these observations, the inventors have developed a parallelentropy encoding scheme that includes a channel for data transferbetween each video section and its right neighbor vertical section. If avideo picture is divided into N vertical sections (N>0), there willgenerally be N−1 channels between neighboring vertical sections. Thesechannels can be used to transfer MB syntax info of the last MB column ofa video section to its right video section to encode the first MB columnof that video section. Based on this structure, the first (e.g.,leftmost) vertical section S0 can be entropy encoded by the masterencoder unit E0 without depending on any other video sections. The slaveencoder units E1 . . . EN can start encoding the other video sections S1. . . SN after the first MB row of each unit's previous vertical sectionis completed. This makes it possible for parallel entropy encoding toproceed. Note that if a channel is allowed from the rightmost verticalsection to the leftmost vertical section for transferring the last MBcolumn's syntax information to the first MB column in a picture, thenthe whole process becomes a sequential encoding process, and one losesthe benefit of parallel encoding. In some codecs, there is a datadependency between the first MB of a given row and the last MB in theprevious row. To avoid this drawback the data dependency between thefirst MB of a given row and the last MB in the previous row can behandled during Stage 2 in order to allow the Stage 1 encoding to be donein parallel.

It is useful for the Stage 2 task of forming the final bitstream to becompleted on a single encoder unit. This is because a slice can crossmultiple video sections, but it must result in a single bitstream. Thatsingle encoder unit (e.g., the master encoder unit E0 or anotherdesignated encoder unit) takes the Stage 1 outputs from each encoderunit E0 . . . EN as its input, and converts them to a single bitstream.To make this happen, there has to be a channel from each encoder unit tothe single encoder unit that forms the final bitstream. It is notedthat, due to the relatively high degree of compression that takes placeduring Stage 1, the amount of data that is transferred over thesechannels is considerably less than the total amount of data for theunencoded vertical sections. This greatly eases the bandwidthrequirements for these data channels.

In some embodiments of the invention, the de-blocking and other postprocessing can be done on the master encoder unit E0 or can be offloadedthis to some other processor. Bandwidth considerations may dictatewhether it is better to do the post processing of the vertical sectionsS0 . . . SN locally on the corresponding encoder units E0 . . . EN or tooffload this process to the master encoder E0 or some other processor.For example, de-blocking generally involves uncompressed pixel data.Therefore, if bandwidth is limited it may be preferable to de-block ofeach vertical section locally with its corresponding encoder unit. Thisavoids having to transfer large amounts of uncompressed data. However,if bandwidth is not an issue, all of the vertical sections could bede-blocked by the master encoder unit E0 or offloaded to some otherprocessor or processors.

3) Stage 1: Parallel Entropy Encoding.

As noted above, during Stage 1, all of the encoder units E0 . . . EN canperform part of the entropy encoding task in parallel. In particular,the bit representation of each symbol within the vertical sections S0 .. . SN can be generated by corresponding encoder units E0 . . . EN.Entropy encoding can only be partially completed for each MB during thisprocess because some syntax element derivation and/or encoding involvesdata dependencies across vertical section boundaries. Some of these datadependencies can be addressed by transferring certain needed data froman encoder unit handling a given section to the encoder unit handling aneighbor of the given section. The data that needs to be transferredfrom an encoder unit handling a given section to the encoder unithandling a neighbor of the given section is referred to herein asboundary syntax data. In the example illustrated in FIG. 2, boundarysyntax data 206 ₀, 206 ₁, . . . 206 _(N-1) for sections S0, S1, . . .SN-1 are transferred from encoder units E0, E1, . . . EN-1 (not shown)to the encoder units E1, E2 (not shown), . . . EN that are encodingtheir immediate next neighbor sections S1, S2 (not shown), . . . SN,respectively.

The content of the boundary syntax data can depend on the specific codecor implementation used. By way of example, and not by way of limitation,for a CAVALC implementation of the AVC standard, boundary syntax datamay include the number of non-zero coefficients for every sub-unit in alast sub-section (e.g., every block in a last macroblock) for a givenvertical section. Alternatively, for a CABAC implementation of the AVCstandard, boundary syntax data may include subsection (e.g., macroblock)syntax elements whose context index increment needs to be derived, e.g.,as specified in AVC standard 9.3.3.1.1, which is incorporated herein byreference.

As also noted above, some data dependency may exist between the lastvertical section SN and the first vertical section S0. Consequently itmay be impractical to implement certain parts of entropy coding entirelyin parallel. By way of example, and not by way of limitation, for CAVLC,the whole macroblock encoding process of skipped macroblocks is skipped.

For CAVLC, in the final output stream, the data for each non-skippedmacroblock includes a symbol called “MB_skip_run”, which is a counter ofthe number of skipped macroblocks between the current macroblock and theprevious non-skipped macroblock. For CAVLC, if a stage 1 encoder setsthe coded bits for a given macroblock to 0, the stage 2 encoder knowsthat this macroblock is skipped. So, there is no need in CAVLC for aseparate flag to indicate whether a macroblock is skipped in the CAVCLcase. For CAVALC, nearly all MB syntax elements can be encoded in Stage1 except for MB_skip_run.

The derivation of MB_skip_run can cross a whole slice and therefore cancross multiple section boundaries. It may be impractical to determine anexact value of MB_skip_run within a vertical section. In such casesMB_skip_run can be encoded in Stage 2. The data needed to computeMB_skip_run is not transferred across vertical section boundaries withthe boundary syntax data, but can instead be transmitted from each slaveencoder unit E1 . . . EN to the master encoder unit E0 as part of outputdata 202 ₀, 202 ₁ . . . 202 _(N) transmitted in the partially encodedbitstreams. The data needed to compute MB_skip_run is not compressed,however it represents a relatively small amount of data compared to thetotal output for a given vertical section. Consequently, transferringthe data needed from each slave encoder unit E1 . . . EN to the encoder(e.g., E0) that encodes MB_skip_run in Stage 2 does not require a largebandwidth. To facilitate computation of MB_skip_run in Stage 2, a skipflag is not transferred. The previous MB skipped flag is not used forstage 1 encoding and is not transferred as part of the boundary data.

For a CABAC implementation, the situation is slight different.Specifically, in CABAC, each macroblock has a symbol called“mb_skip_flag”. For skipped macroblocks only, “mb_skip_flag=1” is codedand all other encoding processes are skipped. For non-skippedmacroblocks, “mb_skip_flag=0” is coded along with other macroblock data.For CABAC, “mb_skip_flag” is encoded in the stage 1 output bin stream.The bin stream is transferred to stage 2 and stage 2 encodes“mb_skip_flag” the same as other MB bins.

Another syntax element that often needs to be derived as part of entropycoding is referred to herein as a subsection QP difference. This syntaxelement refers to a difference in the value of a quantization parameterbetween one subsection (e.g., one macroblock) and the next subsectionthat is to be encoded. The quantization parameter is a setting used bymany codecs, such as H.264, to control the quality of video compression.The quantization parameter regulates the amount of spatial detail thatis saved when picture data is compressed. One example of a subsection QPdifference is known in the H.264 codec as MB_qp_delta, which is adifference in certain macroblock-level quantization parameters. Thesubsection QP difference for each subsection may be encoded in eitherStage 1 or Stage 2, depending on which solution is used. The derivationof the subsection QP difference for a given section may depend on itsprevious subsection's syntax information. If the current subsection isthe first subsection in a row, its subsection QP difference value candepend on the last subsection in the previous subsection row. Whetherthe subsection QP difference can be encoded in stage 1 or stage 2depends on whether this data dependency between the first verticalsection and the last vertical section can be cut off. Examples ofdifferent solutions are described below in section 7) below.

For CABAC implementations of AVC, it is possible to completebinarization for all or part of the subsection (e.g., macroblock) syntaxelements in this stage, and the resulting bin string can be included inthe Stage 1 outputs 202 ₀ . . . 202 _(N). Optionally, the Stage 1encoders can also derive all or part of certain other intermediate datathat can be used in stage 2 encoding, e.g., a context index (ctxIdx) orcontext index increment (ctxIdxInc) for each subsection as additionaloutputs. There are two steps in CABAC encoding. The first step, oftencalled binarization, maps each syntax element to a bin string. Thesecond step is to encode the bin string to a bitstream. There is strongdata dependency among neighboring subsections (e.g., macroblocks) withina slice. A CABAC engine (which determines an offset and range) and acontext model from a context table are used to encode each bin, and areupdated after that bin is encoded. This means the order of bins toencode cannot be changed. Otherwise, an illegal bitstream output wouldbe produced. As a result, step 2 must be processed sequentially for allsubsections in a picture, and cannot be distributed by processingmultiple vertical sections in parallel. In such a case, it is useful forstep 2 to be completed in Stage 2.

By way of example, and not by way of limitation, the derivation ofMB_qp_delta is the same for CABAC as for CAVLC. However, to encodeMB_qp_delta in a CABAC implementation the ctxIdxInc information isneeded. Deriving the ctxIdxInc for a given macroblock requires MB_type,coded block pattern (CBP) and the MB_qp_delta for the previous MB, whichis unknown for the first MB column in the first vertical section.Consequently, in such a case, MB_qp_delta has to be encoded in Stage 2.

4) Stage 2: Encode and Form Final Bitstream

As indicated above Stage 2 encoding can be a sequential process that isimplemented on a single encode unit. This single encoder unit isreferred to herein as the Stage 2 encoder unit. In the example discussedabove, this encoder unit can be the master encoder unit E0. However,embodiments of the present invention are not limited to suchimplementations. The Stage 2 encoder unit may alternatively, be any ofthe slave encoder units E1 . . . EN or a completely separate encoderunit. The Stage 2 encoder unit takes the stage 1 encoding outputs 202 ₀. . . 202 _(N) from each encoder unit as its inputs, and outputs thefinal bitstream 204. The Stage 2 encoder unit may implement any or allof a number of tasks during Stage 2. Examples of such tasks include, butare not limited to: parsing the output of each Stage 1 encoder unit,finishing remaining bitstream encoding tasks, and concatenating theencoded bitstreams in order to form the final bitstream for output. Asused herein, concatenating an encoded bitstream means to append it tothe final bitstream.

The remaining bitstream encoding tasks implemented by the Stage 2encoder may vary depending on the particular codec or implementationused. By way of example and not by way of limitation, for an AVC CAVLCimplementation, the remaining bitstream encoding tasks can include:deriving and encoding MB_skip_run; and, optionally, deriving andencoding MB_qp_delta, depending on which solution is used as describedin section 7 below.

By way of further example and not by way of limitation, for an AVC CABACimplementation, the remaining bitstream encoding tasks can optionallyinclude calculating MB_qp_delta value and its ctxIdxInc value dependingon which solution is used as described in section 7); and encoding binstrings from the Stage 1 outputs to bitstreams.

5) Reducing Bandwidth and Network Delay

Network bandwidth and delay are important factors when consideringoverall encoder performance. For an encoder to achieve betterperformance, it is often desirable to reduce the bandwidth and networkdelays as much as possible. As noted above, there are two types of datachannels in the encoder 200. For convenience, these two different typesof data channels are referred to herein as type A channels and type Bchannels. Type A channels are the data channels for transferringboundary syntax information 206 ₀ . . . 206 _(N-1) between encoder unitsprocessing neighboring vertical sections in one direction, e.g., fromthe encoder unit processing a given vertical section to the encoderprocessing the vertical section to the immediate right of the givenvertical section. Type B data channels are used for transferring Stage 1outputs 202 ₀ . . . 202 _(N) from the encoder units E0 . . . EN to theStage 2 encoder unit. The bandwidth and network delays for type Achannels can be regarded trivial as the data being transferred typicallyonly involves subsection syntax for one column of subsections, e.g., onecolumn of macroblocks. The bandwidth and network delays for type Bchannels are the major concern, as the data being transferred over thesechannels involves all the subsections in a picture. For CAVLC, this isnot a big issue.

The Stage 1 output is primarily the partially encoded bitstream and thebandwidth consumed by this bitstream is not significant due to the highdegree of compression obtained in Stage 1.

For CABAC, the situation is more complicated. There are threepossibilities for handling the Stage 1 output in a CABAC implementation.As depicted in FIG. 3A, a first possibility 300 is for the Stage 1encoders E0 . . . EN to output the bin string only as Stage 1 output301. Then at Stage 2, the Stage 2 encoder can parse the bin string toretrieve all the subsection (e.g., macroblock) syntax elements from theStage 1 output 301 as indicated at 302. The Stage 2 encoder unit canderive ctxIdx for each bin in the bin string as indicated at 304, andencode the bin string to a bitstream as indicated at 306 to produce aStage 2 output bitstream 307. Among the three possibilities, thissolution requires the minimum bandwidth and introduces the smallestnetwork delays. However, this solution comes at the price of introducingthe greatest amount of sequential computation cost of the threepossibilities.

A second possibility 310, illustrated in FIG. 3B, is for the Stage 1encoders E0 . . . EN to output a bin string and a ctxIdx array whichstores ctxIdx for each bin in the bin string as Stage 1 output 311.Then, the Stage 2 encoder unit can directly encode the bitstream fromthe bin string, with the help of ctxIdx array as indicated at 312 toproduce a Stage 2 output bitstream 313. This saves the computation ofbin string parsing and ctxIdx derivation. Thus the sequentialcomputation cost and the overall computation cost is the minimum of thethree possibilities. But the bandwidth requirement and network delay arerelatively high compared to the other two possibilities, as the ctxIdxfor each bin has to be transferred as part of the Stage 1 output 311.

A third possibility 320 shown in FIG. 3C is a tradeoff between the firstand second possibilities. For each subsection, e.g., each MB, the Stage1 output 321 can include the bin string plus a ctxIdxInc packet. Then inStage 2, the Stage 2 encoder unit still does bin string parsing asindicated at 322 and encodes the bin string to a bitstream as indicatedat 328 to provide the stage 2 output 329. However, in thisimplementation, the Stage 2 encoder does not need to derive ctxIdxIncbased on its neighbor MB's syntax info. Instead, it can directlyretrieve the ctxIdxInc from the ctxIdxInc packet, as indicated at 324,and add to ctxIdx Offset to get the ctxIdx for a bin as indicated at326. This saves sequential computation cost compared to the firstpossibility 300. Since only a small portion of the subsection syntaxelements requires ctxIdxInc to calculate ctxIdx, the size of thectxIdxInc packet for one subsection can be quite small, usually lessthan or comparable to the size of the bin string itself. This savesbandwidth and network delay significantly compared to the secondpossibility 310.

6) Solution for Encoding MB_Skip_Run in CAVLC

As discussed above in section 3), in CABAC, the MB_skip_flag is encodedin the first stage the same way as the other macroblock bins.Consequently, there is no further description of skipped macroblockhandling for CABAC cases.

As mentioned above in section 3), the derivation of MB_skip_run in CAVLCinvolves dependencies that cross boundaries between vertical sections.Consequently, it is more practical to derive MB_skip_run in Stage 2during the bitstream combining process. FIG. 4 illustrates a possibleStage 2 method 400 for handling MB_skip_run in a CAVLC implementation.To solve this problem, the MB bitstream size can be transferred alongwith the MB bitstream to the stage 2 encoder unit as part of the Stage 1output 401. In Stage 2, MB_skip_run can be initialized to 0 as indicatedat 402. Then for each incoming MB bitstream, the Stage 2 encoder candetermine whether a given macroblock is skipped by checking the MBbitstream size as indicated at 404. An MB bitstream size of 0 for a giveMB means that the MB is a skipped MB. If the MB bitstream size isdetermined to be zero at 404, the corresponding MB is a skipped MB andthe Stage 2 encoder can update MB_skip_run as indicated at 406 andcontinue to the next MB as indicated at 408. If, at 404 the MB bitstreamsize is determined to be other than zero, the Stage 2 encoder can encodeMB_skip_run as indicated at 410, concatenate the MB bitstream asindicated at 412, and reset MB_skip_run to 0 as indicated at 414 beforeproceeding to the next MB bitstream at 408. In this way, MB_skip_run canbe derived and encoded in Stage 2. It is noted that the foregoing isonly an example of handling a skip run calculation in Stage 2 of aparallel entropy coding method with the scope of embodiments of thepresent invention. This method can be generalized to handling the skiprun for other subsection sizes (e.g., blocks) and is not limited to askip run calculation for macroblocks.

7) Solutions for Encoding MB_QP_Delta (both CAVLC and CABAC)

As mentioned in section 3), a subsection QP difference is used incertain codecs as part of the encoding process. By way of example, andnot by way of limitation, a macroblock quantization parameter,MB_qp_delta is used in both CAVLC and CABAC implementations of AVCcodecs, such as H.264. In such codecs, there are two QP values for eachmacroblock (MB). One quantization parameter is referred to as theencoder assigned QP. The encoder assigned QP may be meaningless, e.g.,if MB_qp_delta does not exist for a macroblock. For example, if a givenmacroblock is skipped or if both CBP==0 and MB_type is not intra 16×16are true, MB_qp_delta does not exist for the given macroblock. In thiscase, its encoder assigned QP would be meaningless. The otherquantization parameter is known as the reconstructed QP. According tothe AVC spec, if MB_qp_delta does not exist for a given macroblock thereconstructed QP for the given macroblock is the same as its previousMB's reconstructed QP. If MB_qp_delta does exist for a given macroblock,the reconstructed QP for the given macroblock is the same as the encoderassigned QP for the given macroblock. MB_qp_delta is specified as thereconstructed QP of the current MB minus the reconstructed QP of itsprevious MB. In other words, the derivation of MB_qp_delta depends onthe previous MB's reconstructed QP. For CABAC, the encoding ofMB_qp_delta also requires the previous MB's syntax info. This introducesthe aforementioned data dependency of the first vertical section S0 onthe last vertical section SN. As mentioned in the end of section 2), forthe sake of performance, it is generally desirable not to add a channelbetween the encoder unit EN for the last video section SN and theencoder unit E0 for the first video section S0. A key feature ofembodiments that avoid such a data channel (and corresponding datadependency) is some solution for encoding the subsection QP difference(e.g., MB_qp_delta) for both CAVLC and CABAC implementations.

By way of example, and not by way of limitation, solutions for encodingthe subsection QP difference include the following solutions referred toas Solution 1 and Solution 2. The flow diagram in FIGS. 5A-5B illustratemethods for implementing Solution 1 in conjunction with parallel entropycoding of the type depicted in FIG. 2.

As shown in FIGS. 5A-5B in Solution 1 the Stage 1 encoders 201 can addan encoder-assigned QP value denoted QP_(E) for each subsection (e.g.,each MB) in the Stage 1 outputs 202 ₀ . . . 202 _(N). Theimplementations of Solution 1 are slightly different for CAVLC andCABAC.

For CAVLC, the MB bitstream 501 is split to two parts referred to as B1and B2, as shown in FIG. 5A. The first part B1 contains the bitstreambefore MB_qp_delta. The second part B2 contains the bitstream afterMB_qp_delta. The Stage 1 encoder 201 does not encode MB_qp_delta ineither of these bitstreams. Similar to the idea described above insection 6) and illustrated in FIG. 4, the Stage 1 encoders 201 sendbitstream sizes along with both bitstream parts B1, B2. For convenience,the bitstream size for the first part B1 is referred to herein as L1,and the bitstream size for the second part B2 is referred to herein asL2. The Stage 1 encoders 201 send both bitstreams B2, B2 and theirrespective bitstream sizes L1, L2 to the Stage 2 encoder unit 203 aspart of the Stage 1 outputs 202 ₀ . . . 202 _(N).

To encode the value of MB_qp_delta, the Stage 2 encoder 203 can firstcompare the bitstream sizes L1, L2 to zero as indicated at 504 and 508.There are three possible cases. In a first possible case, L1 is 0. Thisimplies L2 is also 0. The MB for the bitstreams B1, B2 is therefore askipped MB.

The Stage 2 encoder can simply update MB_skip_run for this MB asindicated at 506 and proceed to the bitstreams for the next MB asindicated at 516.

In a second possible case, the Stage 2 encoder 203 may determine that L1is not zero at 504 but that L2 is zero at 508. In this situation, the MBcorresponding to the bitstreams B1, B2 is not a skipped MB, but there isno MB_qp_delta value for this MB. In this case, the Stage 2 encoder canencode MB_skip_run as indicated at 510, concatenate the first partbitstream B1 as indicated at 512. Since L2 is zero, the second partbitstream B2 is empty so there is no need to go further with the secondpart bitstream B2. Then MB_skip_run can be reset to be 0, as indicatedat 514. The Stage 2 encoder 203 can then proceed to the bitstreams forthe next MB, as indicated at 516.

In a third possible case, the Stage 2 encoder 203 may determine that L1is not zero at 504 and that L2 is also not zero at 508. In thissituation, a reconstructed QP value for the current MB denoted QP_(R) isthe same as the encoder assigned value QP_(E). As indicated at 518, theStage 2 encoder 203 can derive the value of MB_qp_delta as thedifference between the encoder assigned QP value QP_(E) and thereconstructed QP value for the previous MB (denoted QP_(RP)). The Stage2 encoder 203 can update previous reconstructed QP value QP_(RP) to bethe encoder assigned QP value QP_(E) as indicated at 520. The stage 2encoder 203 can encode MB_skip_run and then reset MB_skip_run to 0 asindicated at 522 and concatenate first part bitstream B1 into the Stage2 bitstream 204 as indicated at 524. The Stage 2 encoder can also encodeMB_qp_delta as indicated at 526 and concatenate the second partbitstream B2 into the Stage 2 bitstream 204 as indicated at 528. Whenencoding some syntax elements such as MB_skip_run or MB_qp_delta instage 2, the Stage 2 encoder can append the coded data to the Stage 2output bitstream 204.

In this way, both MB_skip_run and MB_qp_delta can be encoded in stage 2for CAVLC. It is noted that the method depicted in FIG. 5A can beapplied to encoding subsection skip run and subsection QP differencevalues for subsection sizes other than macroblocks in embodiments of thepresent invention.

The flow diagram shown in FIG. 5B illustrates the Solution 1implementation for CABAC. In this implementation, the Stage 1 encoder201 assigns a QP value QP_(E) to MB data 541 as indicated at 502. Asindicated at 542, the Stage 1 encoder 201 also generates a Bin string543 from the MB data 541 but skips MB_qp_delta in the bin string. Asindicated at 544, the Stage 1 encoder 201 also generates a ctxIdxIncpacket 545, but skips the ctxIdxInc corresponding to MB_qp_delta in thectxIdxInc packet. The Stage 1 encoder 201 transfers the QP value QP_(E),the bin string 543, and the ctxIdxInc packet 545 to the Stage 2 encoder203 as indicated at 546.

The stage 2 encoder 203 parses the bin string 543 for a current MB asindicated at 548. During bin string parsing process for the current MB,the Stage 2 encoder 203 can determine whether MB_qp_delta exists for thecurrent MB, as indicated at 550. If at 550 it is determined thatMB_qp_delta does not exist, the Stage 2 encoder 203 can encode the binstring 543 to the Stage 2 output Bitstream 204, as indicated at 560.

If it is determined at 550 that MB_qp_delta does exist, the Stage 2encoder 203 can derive the ctxIdxInc for MB_qp_delta as indicated at 552based on the previous MB's syntax information, which can be madeavailable from the bin string parsing of the previous MB. With thisinformation, the Stage 2 encoder 203 can encode MB_qp_delta, asindicated at 556. The derivation of the value of MB_qp_delta can beimplemented in the same fashion as in the CAVLC implementation describedabove with respect to FIG. 5A. Specifically, as indicated at 554,MB_qp_delta can be derived from the difference between the Stage 1encoder-assigned QP value QP_(E) extracted from the Stage 1 output andthe reconstructed QP value QP_(RP) for the previous MB. The value ofQP_(RP) can be updated to the value of Q_(E), as indicated at 558. TheStage 2 encoder 203 can encode the bin string 543 to the Stage 2 outputBitstream 204, as indicated at 560.

As noted above, there is a second solution for encoding the subsectionQP differences. Solution 2 is to cut off the dependency between theencoding of the first vertical section S0 and the encoding of the lastvertical section SN. By way of example, the Stage 1 encoder 201 or Stage2 encoder 203 can assign a fixed value to the encoder-assigned QP forthe last column of subsections (e.g., the last MB column) in a picture.In this solution the encoder can force each subsection in this lastcolumn to always have a subsection QP difference value syntax element inorder to adjust their QP value to that fixed QP value. In this way, thederivation of the subsection QP difference value for the firstsubsection in each row of subsections for the picture is totallyindependent of the QP value for the last subsection in the previous row.

By way of example, and not by way of limitation, FIGS. 6A-6B illustratemethods 600, 620 for implementing solution 2 in CAVLC and CABACrespectively. As shown in both FIG. 6A and FIG. 6B, the Stage 1 encoderEN for the last vertical section SN can assign fixed value of QP_(E) toeach of the last column macroblocks as indicated at 602, determineMB_qp_delta from the fixed value QP_(E) and the reconstructed QP valueQP_(RP) for the previous MB, as indicated at 604. As shown in FIG. 6A,in the CAVALC implementation, the Stage 1 encoder 201 can encodeMB_qp_delta for the last column of macroblocks to the Stage 1 outputbitstream 202 _(N), as indicated at 606. The Stage 2 encoder 203 canencode MB_skip_run as indicated at 608. By way of example, and not byway of limitation the encoding of MB_skip_run can be implemented, e.g.,as described above in section 6) with respect to FIG. 4.

In the CABAC implementation depicted in FIG. 6B, the Stage 1 encoder canderive MB_qp_delta the same way as for CAVLC and put MB_qp_delta intothe bin string, as indicated at 622 The bin string is sent to the Stage2 encoder 203 as part of the Stage 1 output 202 _(N)

The Stage 2 encoder 203 can derive ctxIdxInc for MB_qp_delta from theStage 1 output as indicated at 624. By way of example, the Stage 2encoder 203 can derive ctxIdxInc for MB_qp_delta if the MB_type, codedblock pattern (CBP) and MB_qp_delta from the previous MB are known.

MB_type generally refers to a syntax element specifying the type ofmacroblock that is being encoded. MB_type can be used to control theencoding of MB_qp_delta. For example, if MB_type is intra 16×16,MB_qp_delta is encoded whether the DCT coefficients are zero or not. Ifthe macroblock is not coded as intra 16×16, MB_qp_delta is only encodedwhen the macroblock has non-zero DCT coefficients.

CBP generally refers to that represents whether coefficient data needsto be encoded in a particular sub-sub-section (e.g., a particular 8×8block within a 16×16 macroblock). It is similar to skip run, but withina subsection.

The processing of ctxIdxInc for MB_qp_delta can be implemented, e.g., asdescribed above with respect to FIG. 5B for the CABAC implementation ofsolution 1. The Stage 2 encoder can then encode the bin string (whichalready includes the bit representation of MB_qp_delta), as indicated at628, to the Stage 2 output bit stream 204. It is noted that the Stage 2encoder 203 can derive ctxIdxInc for MB_qp_delta from the Stage 1 outputduring the process of encoding of the bin string.

In embodiments of the present invention, suitably configured encoderunits can implement Stage 1 and Stage 2 of parallel entropy coding asdescribed above. FIG. 7 illustrates a block diagram of an example of anencoder unit 700 that can implement a part of Stage 1 parallel entropycoding or Stage 2 of parallel entropy encoding or both, as describedabove. The encoder unit may include generally a processor module 701 anda memory 707. The processor module 701 may include one or more processorcores. An example of a processing system that uses multiple processormodules, is a CELL processor, examples of which are described in detail,e.g., in Cell Broadband Engine Architecture, which is available onlineathttp://www-306.ibm.com/chips/techlib/techlib.nsf/techdocs/1AEEE1270EA2776387257060006E61BA/$file/CBEA_(—)01_pub.pdf, which is incorporated hereinby reference.

The memory 707 may be in the form of an integrated circuit, e.g., RAM,DRAM, ROM, and the like. The memory may also be a main memory that isaccessible by all of the processor modules of a multiple processormodule processing system. In some embodiments, the processor module 701may include multiple processor cores 701A, 701B and local memoriesassociated with each core. A Stage 1 coder program 703 and/or Stage 2coder program 705 may be stored in the main memory 707 in the form ofprocessor executable instructions that can be executed on one or morecores of the processor module 701. The Stage coder program 703 and Stage2 coder program 705 may be written in any suitable processor readablelanguage, e.g., C, C++, JAVA, Assembly, MATLAB, FORTRAN and a number ofother languages.

The Stage 1 coder programs 703, 705 may be configured to implementcertain stages of entropy coding during encoding of a video picture intocompressed signal data. By way of example, and not by way of limitation,the Stage 1 coder program 703 may be configured to implement encoding inparallel according to the processes described above. Specifically, theStage 1 coder 703 may include instructions that, when executed perform afirst stage of entropy coding on a vertical section of a video pictureon a row-by-row basis in parallel with encoding of another section byanother encoder unit, which may be configured similar to encoder unit700.

The Stage 2 coder program 705 may include instructions that, whenexecuted by the processor module 701 can generate a final codedbitstream from two or more partially encoded bitstreams produced by twoor more corresponding Stage 1 encoders that execute the Stage 1 coderprogram 703 to produce two or more partially encoded bitstreams.

Data 709 may be stored in the memory 707. Such input data may includebuffered portions of streaming data, e.g., encoded video pictures orsections thereof. During execution of the Stage 1 coder program 703and/or the Stage 2 coder program 705, portions of program code and/orinput data 709 may be loaded into the memory 707 or the local stores ofprocessor cores for parallel processing by multiple processor cores. Byway of example, and not by way of limitation, the data 709 may includedata representing a video picture, or vertical sections thereof, beforeentropy coding, at intermediate stages of entropy coding, or afterentropy coding. These various sections may be stored in one or morebuffers 708, which may be implemented in the memory 707. In particular,sections may be stored in an output picture buffer implemented in thememory 707. The data 709 may also include outputs generated from theStage 1 coder program 703 or Stage 2 coder program 705. Examples ofStage 1 encoder program outputs include, but are not limited to, partialbit streams, bin strings, and syntax elements, such as MB_skip_run,MB_qp_delta, ctxIdx, and ctxIdxInc. Examples of Stage 2 encoder outputsinclude final bitstreams.

The encoder unit 700 may also include well-known support functions 711,such as input/output (I/O) elements 713, power supplies (P/S) 715, aclock (CLK) 717, and cache 719. The encoder unit 700 may optionallyinclude a mass storage device 723 such as a disk drive, CD-ROM drive,tape drive, or the like to store programs and/or data. The encoder unit700 may also optionally include a display unit 725 and user interfaceunit 721 to facilitate interaction between the encoder unit 700 and auser. The display unit 725 may be in the form of a cathode ray tube(CRT) or flat panel screen that displays text, numerals, graphicalsymbols, or images. The user interface 721 may include a keyboard,mouse, joystick, light pen or other device that may be used inconjunction with a graphical user interface (GUI). The encoder unit 700may also include a network interface 727 to enable the device tocommunicate with other devices over a network, such as the Internet.These components may be implemented in hardware, software, or firmwareor some combination of two or more of these.

The processor modules 701, memory 707, support functions 711, userinterface unit 721, display unit 725, and network interface may exchangedata and instructions via a data bus 712.

In certain embodiments the encoder unit 700 may further include an imagecapture unit 729, such as a digital video camera, may be coupled to theprocessor units, e.g., via the I/O elements 713 and the data bus 712.

There are a number of ways to streamline parallel processing withmultiple processors in the encoder unit 700. One example, among othersof a processing system capable of implementing parallel processing onthree or more processors is a CELL processor. There are a number ofprocessor architectures that may be categorized as CELL processors. Byway of example, and without limitation, FIG. 8 illustrates a type ofCELL processor 800 configured to operate as an encoder unit. It is notedthat in certain embodiments, the cell processor 800 may implement bothpart of Stage 1 encoding in addition to Stage 2 encoding.

The CELL processor 800 includes a main memory 802, a single powerprocessor element (PPE) 804, and eight synergistic processor elements(SPE) 806. Alternatively the CELL processor 800 may be configured withany number of SPEs. With respect to FIG. 8, the memory 802, PPE 804, andSPEs 806 can communicate with each other and with an I/O device 808 overa ring-type element interconnect bus 810. The memory 802 may containdata 803 having features in common with the data 709 described above, aStage 1 coder program 809 and/or Stage 2 coder program 811. The Stage 1coder program may have features in common with the Stage 1 coder program703 described above. The Stage 2 coder program 811 may have features incommon with the Stage 2 coder program 705 described above.

At least one of the SPE 806 may include in its local store (LS) codeinstructions 805 and/or a portion of the buffered input data. The codeinstructions may include a portion of the Stage 1 or Stage 2 encoderprogram. If the code instructions 805 include part of the Stage 1encoder program 809, the buffered input data may include unencoded datafor part of a vertical section of a video picture. If the codeinstructions 805 include part of the Stage 2 encoder program 811, thebuffered input data may include one or more partially encoded bitstreamsobtained from one or more Stage 1 encoder units.

The PPE 804 may include in its L1 cache, code instructions 807 havingfeatures in common with the coder program described above. Instructions805 and data 807 may also be stored in memory 802 for access by the SPE806 and PPE 804 when needed. By way of example, and not by way oflimitation, the PPE 804 may be configured (e.g., by suitableprogramming) to divide the Stage 1 or Stage 2 encoder process intomultiple tasks. Each task may include certain data and code foroperating on that data. The PPE 804 may execute some of the tasks andassign other tasks to one or more of the SPE 806.

By way of example, the PPE 804 may be a 64-bit PowerPC Processor Unit(PPU) with associated caches. The PPE 804 may include an optional vectormultimedia extension unit. Each SPE 806 includes a synergistic processorunit (SPU) and a local store (LS). In some implementations, a localstore may have a capacity of, e.g., about 256 kilobytes of memory forcode and data. The SPUs are less complex computational units than PPU,in that they typically do not perform any system management functions.The SPUs may have a single instruction multiple data (SIMD) capabilityand typically process data and initiate any required data transfers(subject to access properties set up by a PPE) in order to perform theirallocated tasks. The SPUs allow the system 800 to implement applicationsthat require a higher computational unit density and can effectively usethe provided instruction set. A significant number of SPEs 806 in asystem, managed by the PPE 804, allows for cost-effective processingover a wide range of applications. By way of example, the CELL processor800 may be characterized by an architecture known as Cell Broadbandengine architecture (CBEA). In CBEA-compliant architecture, multiplePPEs may be combined into a PPE group and multiple SPEs may be combinedinto an SPE group. For the purposes of example, the CELL processor 800is depicted as having only a single SPE group and a single PPE groupwith a single SPE and a single PPE. Alternatively, a CELL processor caninclude multiple groups of power processor elements (PPE groups) andmultiple groups of synergistic processor elements (SPE groups).CBEA-compliant processors are described in detail e.g., in CellBroadband Engine Architecture, which is available online at:http://www-306.ibm.com/chips/techlib/techlib.nsf/techdocs/1AEEE1270EA2776387257060006E61BA/$file/CBEA_(—)01_pub.pdf, which is incorporated hereinby reference.

According to another embodiment, instructions for carrying out parallelentropy coding may be stored in a computer readable storage medium. Byway of example, and not by way of limitation, FIG. 9 illustrates anexample of a computer-readable storage medium 900 in accordance with anembodiment of the present invention. The storage medium 900 containscomputer-readable instructions stored in a format that can be retrievedand interpreted by a computer processing device. By way of example andnot by way of limitation, the computer-readable storage medium 900 maybe a computer-readable memory, such as random access memory (RAM) orread only memory (ROM), a computer readable storage disk for a fixeddisk drive (e.g., a hard disk drive), or a removable disk drive. Inaddition, the computer-readable storage medium 900 may be a flash memorydevice, a computer-readable tape, a CD-ROM, a DVD-ROM, a Blu-Ray,HD-DVD, UMD, or other optical storage medium.

The storage medium 900 contains parallel-encoding instructions 901configured to implement parallel-encoding upon execution by a processor.The parallel-encoding instructions 901 may optionally includeinstructions for identifying encoder units 903, such that a masterencoder unit and its corresponding client encoder units are identifiedin order to process a video stream in parallel. In addition, theparallel-encoding instructions 901 may optionally include instructionsfor partitioning the video stream 905 so that each individual encoderunit may process a partition of the video stream in parallel with all ofthe other encoder units. The parallel-encoding instructions 901 may alsoinclude instructions for performing a mode search on the video streampartitions 907. The mode search may be optionally implemented using thefast intra-mode search and early termination method described incommonly-assigned U.S. patent application Ser. No. 12/553,073, filedSep. 2, 2009, the entire contents of which has been incorporated hereinby reference above.

Additionally the parallel-encoding instructions may include Stage 1entropy coding instructions 909 for implementing Stage 1 entropy codingof vertical sections of a video picture. The parallel-encodinginstructions may also include Stage 2 entropy coding instructions 911for implementing Stage 2 entropy coding on two or more partially encodedStage 1 outputs. The parallel-encoding instructions 901 may alsooptionally include instructions for dealing with errors that may occurduring the encoding process 913. Examples of such error handling aredescribed, e.g., in commonly-assigned U.S. patent application Ser. No.12/553,073, filed Sep. 2, 2009, the entire contents of which has beenincorporated herein by reference above. Lastly, the parallel encodinginstructions 901 may optionally include instructions for compressingvariable length symbols that result from the encoding process 915, e.g.,as described with respect to FIGS. 6A-6B of in commonly-assigned U.S.patent application Ser. No. 12/553,073, filed Sep. 2, 2009, the entirecontents of which has been incorporated herein by reference above.

Embodiments of the present invention allow for more efficient and fasterencoding of digital pictures that take full advantage of thecapabilities of parallel processing capabilities in the entropy codingtask. In certain embodiments and examples described above all three ofmode search, entropy coding and de-blocking are performed in parallelfor different sections of a digital picture with two or more encoderunits. However, in other embodiments one or both of the mode search orde-blocking processes may be performed on a single encoder unit for theentire picture with the remaining one or two processes being performedin parallel for different sections on different encoder units.

Although the present invention has been described in considerable detailwith reference to certain preferred versions thereof, other versions arepossible. For example, although certain embodiments are described inwhich the subsections are identified as macroblocks, embodiments of theinvention may include implementations in which the subsections are otherthan macroblocks. Therefore, the spirit and scope of the appended claimsshould not be limited to the description of the preferred versionscontained herein. Instead, the scope of the invention should bedetermined with reference to the appended claims, along with their fullscope of equivalents.

All the features disclosed in this specification (including anyaccompanying claims, abstract and drawings) may be replaced byalternative features serving the same, equivalent or similar purpose,unless expressly stated otherwise. Thus, unless expressly statedotherwise, each feature disclosed is one example only of a genericseries of equivalent or similar features. Any feature, whether preferredor not, may be combined with any other feature, whether preferred ornot. In the claims that follow, the indefinite article “A”, or “An”refers to a quantity of one or more of the item following the article,except where expressly stated otherwise. Any element in a claim thatdoes not explicitly state “means for” performing a specified function,is not to be interpreted as a “means” or “step” clause as specified in35 USC §112, ¶16. In particular, the use of “step of” in the claimsherein is not intended to invoke the provisions of 35 USC §112, ¶6.

The reader's attention is directed to all papers and documents which arefiled concurrently with this specification and which are open to publicinspection with this specification, and the contents of all such papersand documents incorporated herein by reference.

1. A method for parallel encoding digital pictures, comprising: a)partitioning a digital picture into two or more vertical sections; b)performing a first stage of entropy coding on the two or more verticalsections with two or more corresponding Stage 1 encoder units on arow-by-row basis, wherein the entropy coding of the two or more verticalsections is performed in parallel such that each Stage 1 encoder unitperforms entropy coding on its respective vertical section and returns apartially coded Stage 1 output to a Stage 2 encoder unit, wherein eachpartially coded Stage 1 output includes a representation of data for acorresponding vertical section that has been compressed by a compressionfactor greater than 1; and c) generating a final coded bitstream withthe Stage 2 encoder unit from the partially encoded bitstreams as aStage 2 output.
 2. The method of claim 1 wherein the compression factoris 50 or greater.
 3. The method of claim 1, further comprising, prior tob), performing a mode search on the two or more vertical sections on arow-by-row basis.
 4. The method of claim 1, wherein b) includestransferring boundary syntax data for a last sub-section of a row of onevertical section to an encoder that performs entropy coding for avertical section that borders on the last sub-section.
 5. The method ofclaim 4, wherein b) includes entropy coding using context adaptivevariable length coding and the boundary syntax data includes a number ofnon-zero coefficients for every block.
 6. The method of claim 4 whereinb) includes entropy coding using context adaptive binary arithmeticcoding and the boundary syntax data includes one or more syntax elementsfrom which a context index increment can be derived.
 7. The method ofclaim 1, wherein b) and c) include entropy coding using context adaptivevariable length coding and wherein b) and c) include avoiding datadependency between Stage 1 encoders in encoding a subsection skip run.8. The method of claim 7, wherein the partially coded Stage 1 outputgenerated by each Stage 1 encoder unit includes a bitstream containingpartially encoded bits representing each subsection in the verticalsection encoded by the given encoder unit, and a corresponding size ofthe bitstream.
 9. The method of claim 8, wherein, during c), the Stage 2encoder initializes a subsection skip run to 0 as indicated and, foreach incoming MB bitstream, the Stage 2 encoder determines whether agiven macroblock is skipped by checking the size of the bitstream. 10.The method of claim 9, wherein, if the bitstream size is determined tobe zero, the corresponding subsection is a skipped subsection and theStage 2 encoder updates the value of the subsection skip run, andwherein, if, the size of the bitstream size is determined to be otherthan zero, the Stage 2 encoder encode the subsection skip run,concatenate the bitstream, and reset subsection skip run to zero. 11.The method of claim 1 wherein b) and c) include entropy coding usingcontext adaptive binary arithmetic coding and the partially coded Stage1 output generated by each Stage 1 encoder unit includes a binary stringcontaining a plurality of bins representing each subsection in thevertical section encoded by the given encoder unit.
 12. The method ofclaim 11, wherein the Stage 1 encoders output the binary string only asthe Stage 1 output, and wherein, during c), the Stage 2 encoder parsesthe binary string to retrieve all subsection syntax elements from theStage 1 output and derives a context index for each bin in the binstring, and encode the bin string to the final coded bitstream.
 13. Themethod of claim 11 wherein the partially coded Stage 1 output generatedby each Stage 1 encoder unit further includes a context index array thatstores a context index for each bin in the bin string.
 14. The method ofclaim 13, wherein the Stage 1 encoders output the binary string and thecontext index array as the Stage 1 output and, during c), the Stage 2encoder unit directly encodes the final coded bitstream from the binstring and the context index array.
 15. The method of claim 11 whereinthe partially coded Stage 1 output generated by each Stage 1 encoderunit further includes a context index increment packet that includes acontext index increment for one or more bins in the bin string thatrequire a context index increment to derive their context index.
 16. Themethod of claim 15 wherein, during c), the Stage 2 encoder unit parseseach bin string, retrieves a corresponding context index increment fromthe context index increment packet and adds the context index incrementto a context index Offset to determine the context index for a bin. 17.The method of claim 1, wherein b) and c) include avoiding datadependency between Stage 1 encoders for first and last vertical sectionsin encoding a subsection quantization parameter (QP) difference.
 18. Themethod of claim 17, wherein b) and c) include entropy coding usingcontext adaptive variable length coding and wherein the Stage 1 outputfor a given subsection includes an encoder-assigned QP value (QP_(E))for the given subsection, a first part containing a bitstream before thesubsection QP difference, and a second part containing a bitstream afterthe subsection QP difference and wherein the subsection QP difference isnot encoded in either the first or second parts, and wherein the Stage 1output includes a first size for the first part and a second size forthe second part.
 19. The method of claim 18, wherein during c) the Stage2 encoder unit compares the first and second sizes to zero, and wherein:if the first size is zero, the Stage 2 encoder updates a subsection skiprun for the given subsection, if the first size is not zero and thesecond size is zero, the Stage 2 encoder encodes the subsection skiprun, concatenates the first part and resets the subsection skip run tozero, if both the first and second sizes are not zero, the Stage 2encoder derives a value of the subsection QP difference as a differencebetween the encoder assigned QP value QP_(E) and a reconstructed QPvalue for a previous subsection, updates the reconstructed QP value forthe previous subsection with the encoder assigned QP value QP_(E),encodes the subsection skip run and then sets the subsection skip run tozero, concatenates the first part into the Stage 2 Output, encodes thevalue of the subsection QP difference, and concatenates the second partinto the Stage 2 Output.
 20. The method of claim 17) wherein b) and c)include entropy coding using context adaptive binary arithmetic codingand wherein the Stage 1 output for a given subsection includes anencoder-assigned QP value (QP_(E)) for the given subsection, a binarystring for the given subsection that does not include a value of asubsection QP difference, and a context index increment packet that doesnot include a context index increment corresponding to the subsection QPdifference.
 21. The method of claim 20 wherein during c), the stage 2encoder parses the binary string and determines whether a subsection QPdifference exists for the given subsection, wherein, if the subsectionQP difference does not exist, the Stage 2 encoder encodes the binarystring to the Stage 2 Output, if the subsection QP difference doesexist, the Stage 2 encoder encodes the binary string and, duringencoding of the binary string, the Stage 2 encoder derives the contextindex increment for the subsection QP difference based on a previoussubsection's syntax information and encodes the subsection QP differenceusing the derived context index increment.
 22. The method of claim 17wherein b) and c) include cutting off a data dependency encoding of afirst vertical section and encoding of a last vertical section.
 23. Themethod of claim 22 wherein, for each subsection in a last column ofsubsections, the Stage 1 encoder assigns a fixed value to anencoder-assigned QP value (QP_(E)) and forces each subsection in thelast column to always have a subsection QP difference value syntaxelement.
 24. The method of claim 23 wherein the Stage 1 encoder assignsthe fixed value and determines a subsection QP difference value for eachsubsection in the last column from the fixed value and a reconstructedQP value for a next-to-last column.
 25. The method of claim 24, whereinb) and c) include entropy coding using context adaptive variable lengthcoding, and wherein the Stage 1 encoder encodes the subsection QPdifference value and wherein the Stage 2 encoder encodes a value of asubsection skip run from the Stage 1 output.
 26. The method of claim 24,wherein b) and c) include entropy coding using context adaptive binaryarithmetic coding, wherein the partially coded Stage 1 output generatedby each Stage 1 encoder unit includes a binary string containing aplurality of bins representing each subsection in the vertical sectionencoded by the given encoder unit, and wherein the Stage 1 encoderincludes determined subsection QP difference value into the binarystring.
 27. The method of claim 26 wherein the Stage 2 encoder derives acontext index increment for the subsection QP difference from the Stage1 output and encodes the subsection QP difference value and the binarystring to the Stage 2 output.
 28. A computer-readable storage mediumhaving computer executable program instructions embodied therein,wherein the computer executable program instructions are configured,when executed to: a) partition a digital picture into two or morevertical sections; b) perform a first stage of entropy coding on the twoor more vertical sections with two or more corresponding Stage 1 encoderunits on a row-by-row basis, wherein the entropy coding of the two ormore vertical sections is performed in parallel such that each Stage 1encoder unit performs entropy coding on its respective vertical sectionand returns a partially coded Stage 1 output to a Stage 2 encoder unit,wherein each partially coded Stage 1 output includes a representation ofdata for a corresponding vertical section that has been compressed by acompression factor greater than 1; and c) generate a final codedbitstream with the Stage 2 encoder unit from the partially encodedbitstreams as a Stage 2 output.
 29. A system for parallel digitalpicture encoding, comprising: two or more Stage 1 encoder units; a Stage2 encoder unit; a memory coupled to the Stage 1 and Stage 2 encoderunits; instructions embodied in the memory and executable by theprocessor, wherein the instructions are configured to implement a methodfor the parallel encoding of the one or more digital pictures, whereinthe computer executable program instructions are configured, whenexecuted to: a) partition a digital picture into two or more verticalsections; b) perform a first stage of entropy coding on the two or morevertical sections with two or more corresponding Stage 1 encoder unitson a row-by-row basis, wherein the entropy coding of the two or morevertical sections is performed in parallel such that each Stage 1encoder unit performs entropy coding on its respective vertical sectionand returns a partially coded Stage 1 output to the Stage 2 encoderunit, wherein each partially coded Stage 1 output includes arepresentation of data for a corresponding vertical section that hasbeen compressed by a compression factor greater than 1; and c) generatea final coded bitstream with the Stage 2 encoder unit from the partiallyencoded bitstreams as a Stage 2 output.